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The second of four symposium papers brings together 
the results of four separate statistical analyses of findings on 
computer-based instruction (CBI) in 199 comparative studies, of which 
32 were conducted in elementary schools, 42 in high schools, 101 in 
universities and colleges, and 24 in adult education settings. Each 
was a controlled, quantitative study which met predefined standards 
for methodological adequacy. The following results and study features 
were observed: (1) students generally learned more in classes in 
which they received CBI; (2) students learned their lessons with less 
instructional time in classes in which they received CBI; (3) 
students liked classes more when they received computer help; (4) 
students developed more positive attitudes toward computers when they 
received help from them in school; (5) computers had little effect in 
the areas of course withdrawal and attitude toward subject matter; 
(6) study results were consistently stronger in published studies 
than in those that were not published; (7) effects were greater when 
different teachers taught experimental and control groups; (8) 
effects tended to be greater in more recent and in short studies; and 
(9) effects were somewhat greater in studies reported in journal 
articles than in dissertations. It is suggested that editorial 
screening, experimental design flaws, and instructional quality may 
have contributed to this favorable picture of CBI. The four 
meta-analyses are listed as references. (MES) 
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Many people believe that computer technology will change society in the 
years ahead as completely as the invention of the printing press did 500 years ago 
or as the invention of writing did thousands of years ago. These earlier inventions 
gave people new ways of encoding, storing, and retrieving information, and they 
ultimately changed the way people worked, the way they played, and probably 
even the way they thought. Computers have also given us a radically different 
way of hauidling information, and so it seems inevitable that they too will 
dramatically alter the way we lead our lives. 

Educational researchers and developers therefore are no longer asking 
whether a computer revolution will occur in education. They are asking instead 
how it will occur. Will the changes in education come swiftl} and smoothly, or will 
education's transition to the computer age be full of false starts and costly 
mistakes? How long will it take for educators to start usmg the computer well? 

During the past two decades, hundreds of educators and evaluators have 
examined the effectiveness of programs of computer-based teaching. We joined this 
effort because we believe that adapting to the computer age is one of the major 
challenges facing schools today and that evaluation studies can help schools meet 
this challenge. Our approach was to analyze statistically, or meta-analyze, 
findings from as many evaluation studies of computer-based instruction as we could 
find. Our purpose was to provide teachers, researchers, and policy makers with an 
overview of what educators have accomplished to date with computer-based 
instruction. 

We carried out four separate statistical r.l tir..i.Mv>v cuniputor- 

based instruction (Bangert-Drowns, Kulik, & KuIik, jW.j, i:, Kuiik ^ Kulik, in 
press; C. Kulik, Kulik, & Shwalb, 1986; J. Kulik, Kulik, & Bangert-Drowns, 1985). 
These analyses covered a total of 199 comparative studies: 32 in elementary 
schools; 42 in high schools; 101 in universities and colleges; and 24 in adult 
education settings. Each of the 199 studies included in our analyses was a 
controlled, quantitative study that met our predefined standards for methodological 
adequacy. The studies covered use of the computer m (a) computer-assisted 
instruction, or CAI, including drill-and-practice and tutorial instruction; (b) 
computer-managed instruction, or CMI; and (c) computer-enriched instruction, or 
CEI, including the use of the computer as a calculating device, programming tool, 
and simulator. 



Most of the studies reported that computer-based instruction has positive 
effects on students. 

1. Students generally learned more in classes in which they received computer- 
based instruction. The average effect of computer-based instruction in ail 
199 studies was to raise examination scores by 0,31 standard deviations, or 
from the 50th to the 61st percentile. 

2. Students also learned their lessons with less instructional time. The average 
reduction in instructional time in 28 investigations of this point was 32%. 

3. Students also Hked their classes more when they received computer help. 
The average effect of computer-based instruction in 17 studies was to raise 
attitude-toward-instruction scores by 0.28 standard deviations. 
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4. Students developed more positive attitudes toward computers when they 
received help from them in school. The average effect size in 17 studies on 
attitude toward computers was 0.33. 

5. Computers did not, however, have positive effects in every area in which 
they were studied. The average effect of computer-based instruction in 29 
studies of attitude toward subject matter was near zero, and the average 
effect was also near zero in 23 studies of course withdrawals. 

Study Features and Outcomes 

A few study features were consistently related to outcomes of computer- 
based education. 

• 

1. Study results were consistently stronger in published studies and weaker in 
unpublished ones (p < .01). The average effect of computer-based 
instruction in published studies was to raise student examination scores by 
0.46 standard deviations, whereas its average effect in unpublished studies 
was to raise scores by only 0.23 standard deviations. 

2. Effects were larger when different teachers taught the experimental and 
control groups (p < .05). Effects were smaller when the same teacher was 
responsible for both groups. With the same teacher in charge of 
experimental and control groups, average size of effect on examination 
scores was 0.24 standard deviations. With different teachers in charge of 
tlie groups, the average effect was 0.40 standard deviations. 

3. Effects tended to be larger in more recent studies and smaller in older 
studies (p < .05). The average effect of computer-based instruction in 
studies published before 1975 was to raise examination scores by 0.24 
standard deviations; the average effect in studies published in later years 
was a score increase of 0.36 standard deviations. 

4. Effects were also somewhat larger in short studies and weaker in longer 
ones (.10 < p < .20). The average effect of computer-based instruction in 
short studies was to raise examination scores by 0.36 standard deviations, 
whereas its average effect in longer studies was to raise scores by 0.27 
standard deviations. 

Because study features were moderately intercorrelated, multiple regression 
analyses were carried out on study feature data. In the multiple regression 
equation developed from the full data set, three of the four study factors had 
significant weights: publication source, control for instructor effects, and study 
duration. The regression weight for the fourth study feature— study year— reached 
a borderline level of significance. 

Discussion 

Why have evaluations of computer-based instruction produced such positive 
results? Several different factors might have contributed to the favorable picture 
in the literature: 

Editorial gatekeeping. Journal editx)rs and reviewers may prefer to publish 
strong and significant results rather than weak and insignificant ones. 
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Experimental design flaws: Design flaws in evaluation studies may allow 
researcher biases and expectations to color study results. 

Instructional quality: The positive results from meta-analytic studies may 
reflect real differences in the quality of conventional and computer-based 
instruction. 

Editorial Gatekeeping 

If editorial gatekeepers base their publication decisions on the significance of 
study findings rather than on study quality, then published studies provide a 
distorted picture of what actually works in education. In such a case, an educator 
could get a better picture of what works from the clearinghouse literature, the 
dissertation literature, cr— better yet—the file-drawer and wastebasket 
literature. The poorest guide to what works would b€ the most highly peer- 
reviewed literature. 

Before throwing away the most respected literature in education, however, 
we should consider another possibility. The difference in results in published and 
unpublished reports may have another cause. We should remember that the 
authors of joiumal and dissertation studies are different individuals working under 
different circumstances. They differ in their research experience, in their 
resources, in their relationship to instructional developers, and in many other 
respects. Such differences can explain— just as well as editorial gatekeeping can— 
the differences in results found in dissertations and journals. It seems to us that 
we know too little about what lies behind the difference in journal and dissertation 
results to reject out-of-h and either kind of result. 

Experimental Design 

Can flaws ir experimental design explain— or explain away— the positive 
findings from studies of computer-based instruction? Some reviewers think so. 
They believe that with imperfectly controlled experiments, positive results are 
more likely to occur than negative ones. Among the factors that might distort 
results in an imperfectly controlled evaluation are differences in time-on-task, self- 
selection differences in assignment to comparison groups, and uncontrolled teacher 
effects. 

The evidence from our meta-analyses is that not all such factors are 
important. The positive results of evaluations of computer-based instruction cannot 
be attributed to differential time-on-task for comparison groups, for example. 
Studies that control for time-on-task have produced nearly the same results as 
studies without strict controls on instructional time. Actual records of instructional 
time have been collected in several studies, and these records suggest that students 
in computer groups often receive instruction for shorter periods than conventional 
students do. 

Probably no other methodological point has received as much attention in 
evaluation research in recent years as the distinction between random experiments 
and quasi-experiments. Random experiments are generally thought co prodi e 
clear and consistent results; quasi-experiments are often thought to produje 
inconsistent and biased results. In our meta-analyses, random experiments and 
quasi-experimentc produced the same results. Our meta-analytic results did not 
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support the idea that the nature of subject assignment to groups is an important 
methodological flaw in evaluation studies of computer-based instruction. 

Results from studies with and without controls for instructor effecus are 
somewhat different, however. In the typical study with the same instructor 
teaching experimental and control classes, the effect of computer-based teaching 
seamed modest. In the typical study with different mstructors in experimental and 
control classes, the effect of computer-based teaching seemed more substantial. 

Why should one-instructor and two-instructor experiments produce 
somewhat different results? It is not at all obvious to us. It may be, for example, 
that in two-instructor experiments, the poorer instructor is usually assigned to the 
control condition and the better instructor to the experimental condition, and the 
difference between conditions is magnified because of these teacher assignments. If 
this is the case, then one-instructor studies more accurately assess the effects of 
computer-based instruction. It may also be, however, that in one-instructor studies 
there is diffusion of the innovative treatment to the control condition. Involvement 
of a teacher in an iiiaovative approach to instruction m^^y have a general effect on 
the quality of the instnictor's teaching. Outlining objectives, constructing lessons, 
preparing evaluation materials, and working with computer materials- 
requirements in computer-based instruction— may help a teacher to do a better job 
in a conventional teaching assignment. If this is the case, two-instructor studies 
provide the better basis for estimating the sl^e of an experimental effect. 

Jnstructioncil Design 

Computer-based instruction is often wel -designed instruction. The hard work of 
an instructional design teams often ensures the quality of computer materials. 
Objectives are usually clear and explicit. Instruction is carefully sequenced. The 
materials engage the learner's attention and encourage learner activity. The 
program provides frequent feedback to the learner. Instructional design teams 
often spend 100 hours developing just one hour of computer lessons. 

Certain features of the computer make it an especially attractive medium for 
instructional designers. Compute 3 can generate attractive and complex graphics 
quickly. Computers can simulate motion. They can give undivided attention to a 
single learner. They can provide complex evaluations of a learner's performance. 
They can wait patiently. They can be programmed to model a learner's cognitive 
processes. 

In certahi respects, computer lessons seem to have an advantage over 
lessons presented by classroom teachers. Few classroom teachers can put 100 
hours of preparation time into each one-hour lesson. Classroom teachers cannot 
give each individual in a large classroom their continuous, undivided attention. 
Classroom teachers can be notoriously slow at grading student work and preparing 
reports. And their patience is often tried by their students. 

Can such differences account for the superior record of computer-based 
instruction in evaluation studies? They might. It is possible that tiie computer has 
farc'i so well in evaluation studies because programs of computer-based instruction 
have generally been well-designed, and computers have delivered instruction in an 
attractive and engaging v^ay. It is possible, in other words, that we should take 
the findings of studies of computer-based instruction at face value, and conclude 



ERIC 7 ZZ 



20 

CBI: What Evaluation Say 



5 



that the computer has so far been an excellent vehicle for the delivery of 
instruction. 

Conclusions 

Among the conclusions that can be drawn from our analysis, the following 
three seem especially important to me: 

1. Most programs of computer-based instruction evaluated in the past have 
produced positive effects on studen^ learning and attitudes. Future 
programs for developing and implementing computer-based instruction 
should therefore be encouraged. If such programs are as carefully designed 
as current programs are, they will most likely produce positive results. 

2. Both journal articles and dissertations present a basically positive picture of 
results of computer-based instruction, but the findings reported in journa! 
articles are clearly more favorable. Researchers should give high p^'iority to 
finding out what factors produce differences in journal and dissertation 
results. Does editorial gatekeeping lead professional journals to present a 
distorted picture of social science findings? Or do dissertation authors simply 
measure experimental effects less well tlian do more seasoned researchers? 

3. Although a variety of different research designs can be used to show the 
effectiveness tf computer-based instruction, certain research designs seem to 
produce more positive results. Studies where the same instructor teaches 
both experimental and control classes, for example, report somewhat weaker 
effects than do studies with different experimental and control teachers. 
Str.dibo of long duration often report weaiker effects than do short studies. 
Reasons for the difference in results from studies using different 
experimental designs are not well understood, however. Research on such 
factors should be encouraged. 
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